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ABSTRACT 


Partners Advancing Childhood Education (PACE) is an Investing in Innovation (i3) development 
grant funded by the Office of Innovation and Improvement, U.S. Department of Education. PACE is 
a family engagement program that enhances family engagement skills among educators and 
leadership skills among parents to support the academic and social-emotional development of 
children in pre-K through third grade (ages three to eight). The impact evaluation used a quasi- 
experimental design (QED) to examine the effect of PACE participants’ mathematics, language, 
literacy, and social emotional development. PACE enrolled three cohorts of students over 3 years 
(cohort 1: 2014; cohort 2: 2015; cohort 3: 2016) and followed each cohort over a full year 
(September through September). Outcomes for PACE students were compared to a sample of 
students with similar characteristics who participated in business-as-usual, traditional academic 
instruction in matched pre-K and elementary programs. Comparison students were recruited 
using similar methods and time frames as those used for PACE students. Using hierarchical linear 
modeling, we found no statistically significant difference in mathematics, language, literacy, or 
social emotional development gains among PACE students compared to those exposed to business- 
as-usual. 
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1. INTRODUCTION 


Parent engagement, defined as “parents and school staff working together to support and improve 
the learning, development, and health of children and adolescents” (Centers for Disease Control 
and Prevention, 2015) has been found to be an important part of increasing minority students’ 
academic achievement (Jeynes, 2003). Students with more involved parents have also been shown 
to have better behavior and enhanced social skills (E] Nokali, Bachman, & Votruba-Drzal, 2010). 
Research indicates that low-income families (Hill & Taylor, 2004; Lee & Bowen, 2006) and families 
living in high-poverty environments (Waanders, Mendez, & Downer, 2007) may benefit from 
family engagement programs. However, there is a need for an increased number of evidence-based 
family engagement programs for children as they transition from pre-K to early childhood 
education. To fill this gap, in 2014, United Way of Greater Atlanta (UWGA) was awarded an 
Investing in Innovation (i3) grant for Partners Advancing Childhood Education (PACE), a family 
engagement program for low-income families with children ages three to eight and the educators 
that support them. PACE is an iteration of SPARK (a W.K. Kellogg Foundation national school 
readiness program) that focused on children ages zero to five and further integrated United Way 
programs into the curriculum, including Parents as Teachers and the K-3 Incredible Years. The 
purpose of this study is to evaluate the expansion of the SPARK and PACE models into a 
comprehensive program for families with children ages three to eight. 


1.1. Program Description 


The i3 PACE program is a 2-year family engagement program that aims to improve educational 
and social-emotional outcomes of children ages three to eight by strengthening family engagement 
skills among educators and leadership skills among parents. The i3 PACE program builds upon two 
United Way of Greater Atlanta (UWGA) early-childhood education programs for children ages zero 
to six, SPARK and PACE. SPARK is a W.K. Kellogg Foundation national school readiness program 
targeting children ages three through six at risk for being inadequately equipped to succeed in 
elementary school. PACE was an iteration of SPARK that focused on children ages zero to five and 
further integrated UWGA programs into the curriculum, including Parents as Teachers and 
Incredible Years. In prior evaluations, SPARK and PACE benefitted children and their parents. 
Comparisons of SPARK cohorts with control groups show that SPARK children outperformed their 
peers on standardized tests in Grades 1 and 2 and had significantly better school attendance. A 
2012-13 PACE evaluation showed both child educational and parent knowledge gains. 


Similar to SPARK and PACE, the i3 PACE model encompasses five key components targeted toward 
parents, children, and educators. The key components include personal visitation for parents and 
students, school transition councils for educators and parents, parent group activities, 
professional development for educators, and a summer transition academy for students (Table 1). 


National Evaluation of i3 05/16/2016 
2 


Inputs 


Community 
Partnerships 


DeKalb County 
Schools 

Atlanta Speech 
School 

Georgia Department 
of Early Care and 
Learning 

UGA Center for 
Family Research 
PAT National Center 
Parent Services 
Project 


Curriculum 


Parents as Teachers 
Incredible Years 
Stronger Together 


Personnel 


Parent Educators 
(PreK and School 
Age) 

UWGA Leadership 
Team 

School 
Administration and 
K-3 Teachers 


Table 1. PACE Logic Model 


Key Components Outputs 
Personal Visitation e % of children that receive all instruction categories 
e 24 monthly personal visits |e % of Parent Educators observed by a supervisor 
e Health and Developmental |e % of children who receive 3+ Personal Visits 
Screenings e % of enrolled children who receive 1+ Personal Visits 
e Referral Resource Network | e _% of parents who report satisfaction with Personal 
Visit experiences on personal visit parent survey 
Parent Group Meetings e % of group connection sites that receive all instruction 
e 18 monthly group categories 
connection meetings e % of group connection sites that hold 5+ meetings 
e 6monthly family e % of parents attending monthly group meetings 
leadership institute (FLI) |e % of parents who report positive outcomes on group 
meetings connection parent survey 
e Parent Led Community e % of parents who report positive outcomes on FLI 
Project survey 
e % of parents who attend at least 1 FLI meeting 
e % of parents who attend 4+ FLI meetings 
e % of parents who report satisfaction on FLI survey 
School Professional e % of professionals who report positive outcomes on 
Development (PD) PD survey 
e Professional Development e  % of professionals who report positive outcomes on 
events to support Family summer transition academy survey 
Engagement e # of PD events held for professionals 
e  # of PD events held for STA teachers 
e % of administrative professionals who attend PD 
events 
e % of educational professionals who attend PD events 
e % of childcare professionals who attend PD events 
e % of professionals who report satisfaction on PD 
survey 
School Transition Council e % of councils implementing school transition activities 
(STC) ¢ %of meetings involving diverse representatives 
e Quarterly meetings e % of councils implementing 2+ meetings 
e Year-round School ¢ % of members attending at 1+ meetings 
Transition Activities ¢ % of members who report satisfaction on STC survey 
Summer Transition Academy | e Ratio of teachers to students 
(STA) e %of enrolled students who attend 15+ days of STA 
e 4weeks,5 daysperweek |e % of PACE families enrolled in STA 
e % of parents who report satisfaction with the STA 
e % of educators who report satisfaction on STA survey 


Short-Term Outcomes 


Ready Parents: Improve 
Parent and Family 
Engagement in Child’s 
Education 


Increase knowledge 
of child development 
Increase in healthy 
home indicators 
Increase in school 
involvement 
Increase in 
leadership behaviors 
Increase in # of 
identified families 
receiving referral 
services 


Ready Schools: Build 
Family-School 
Partnerships 


Increase in educator 
participation in 
school transition 
activities 

Increase in # of 
school transition 
activities offered 


Long-Term Outcomes 


Ready Students: 
Increase School 
Readiness and 
Academic 
Achievement 


Increase in 
language skills and 
performance 
Increase in literacy 
skills and 
performance 
Increase in Math 
skills and 
performance 
Increase in social 
skills and positive 
school behaviors 
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1.1.1 Personal Visitation 


UWGA parent educators provided families with 24 monthly personal visits over a 2-year 
program cycle. Taking guidance from the United Way of Greater Atlanta pre-K Parents as 
Teachers and the K-3 Incredible Years programs, parent educators worked with students and 
their guardians to improve student academic performance and whole-child development 
through one-on-one activities with the child and hands-on activities for parents and children 
to practice during the month. Additionally, the parent educators worked with families to 
establish healthy home routines to support child development and academic success. Data 
were collected during the first two sessions and then every 12 months. 


1.1.2 Parent Group Meetings and Family Leadership Institute 


UWGA parent educators invited parents to participate in monthly meetings. Each monthly 
parent meeting provided parents with an opportunity to network with other parents in their 
community, review and discuss educational material related to healthy home routines for 
child development and parent leadership in schools, and practice their new skills to support 
their use at home. 


UWGA created two venues to facilitate parent group meetings: Group Connections and the 
Family Leadership Institute (FLI). Group Connections were designed to train and support all 
PACE parent/guardians to be leaders in early childhood and school settings. These meetings 
took place monthly and were held at each PACE treatment school. The Family Leadership 
Institute (FLI) is a series of six meetings based on the FLI curriculum that is more time- 
intensive and targeted for parents/guardians who exhibit promising leadership skills or 
have indicated a desire to be community or school leaders. Given the significant investment 
of time that FLI participants must make to each session, only a minority of parents were 
ready to attend the FLI. Those parents practiced and shared their leadership skills with other 
parents in the other key components of PACE. 


1.1.3 School Transition Council 


School administration, teachers, and early childhood educators in the community as well as 
PACE parents of pre-K-Grade 3 students were invited to participate in the School Transition 
Council at their local elementary school. The School Transition Council met quarterly each 
year to discuss current school transition policies and plan school transition activities 
throughout the year. In addition to council meetings, UWGA provided elementary school staff 
and early childhood educators in the community with professional development 
opportunities around supporting parent engagement in education and school transitions. 
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1.1.4 Summer Transition Academy 


Each summer, rising kindergarten to rising Grade 3 students identified as struggling readers 
were invited to participate in the Summer Transition Academy (STA). Using data collected 
through PACE personal visits, as well as school-level data provided by elementary school 
teachers, PACE students who performed just below grade-level standards and would likely 
achieve the grade-level literacy scores with additional support were nominated for inclusion 
in the STA. The STA was offered five half-days a week for 4 weeks, and could host up to 80 
students, with two classes of 10 students per grade level. In collaboration with Atlanta 
Speech School and the Georgia Grade Level Reading Campaign, the STA provided additional 
academic support for students to enhance reading comprehension, fluency, and vocabulary. 


1.1.5 Professional Development 


UWGA provided elementary school staff and early childhood educators in the community 
with professional development opportunities around supporting parent engagement in 
education and school transitions. Professional development events included an initial 
training focused on the Stronger Together curriculum, preparatory training for educators 
involved in the Summer Transition Academy, and training for educators involved in the 
School Transition Council on how to develop and implement school transition events. 


2. IMPACT STUDY DESIGN 


This impact study used a 1-year, pre-post quasi-experimental design (QED) with a 
comparison group to examine the impact of PACE’s parent engagement and family-school 
partnerships on the academic and social-emotional outcomes of students aged three through 
eight in select DeKalb County Schools compared to business-as-usual. Specifically, we 
examined the impact of exposure to PACE on changes in 1) math achievement, 2) reading 
achievement, 3) language achievement, and 4) social-emotional development, for students 
after one program year. Changes in outcomes were measured using recognized assessments 
with proven reliability and validity. To measure social emotional development, all parents 
responded to questions about their children’s social skills and problem behaviors via the 
Social Skills Improvement System (SSIS) (Gresham & Elliott, 2008). Math, language, and 
literacy growth were measured using different assessments according to participants’ grade 
and time assessed. The Bracken School Readiness Assessment (BRSA) (Panter & Bracken, 
2009) was used to measure literacy and mathematics development among children who 
entered the PACE program in PreK and Kindergarten. The Peabody Picture Vocabulary Test 
(PPVT) (Dunn & Dunn, 2007) was used to measure language development among children 
in PreK and Kindergarten. Both the BRSA and PPVT were administered in English and 
Spanish languages by UWGA staff. Children who entered the PACE program in Grades 1 and 
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2 were assessed on mathematics, literacy, and language development using the Iowa Test of 
Basic Skills (ITBS) and Measures of Academic Progress (MAP) depending on their grade level 
and assessment year. Both tests were administered by the DeKalb County School District and 
changes to the testing calendar resulted in children in grades 1 and 3 taking the IOWA exam 
in 2014 and 2015 and children in grades 1-5 taking the MAP exam in 2016-2018. The study 
sample included cohorts 1, 2 and 3 which were recruited at the beginning of the 2014, 2015, 
and 2016 school years, respectively. Students were followed over one calendar year. The 
three cohorts consisted of 685 children (57% PACE; 43% comparison). Outcomes for PACE 
students who participated in PACE services within the first year of enrollment were 
compared to students with similar characteristics who participated in business-as-usual 
traditional academic instruction in matched pre-K and elementary programs. 


Participation in PACE was defined as having participated in at least one of the following: 


e Two personal visits (to allow for data collection in the first visit and the provision of 
PACE services thereafter); 

e One Group Connection event; 

e One or more days of the Summer Transition Academy; and/or 

e One day of Family Leadership Institute training. 


Comparison students were recruited using similar methods and time frames as those used 
for PACE students. Propensity score matching was used to predict the selection of families 
into the PACE program and baseline equivalency was established separately for each of the 
four domains examined. For each outcome domain, hierarchical linear modeling 
(Raudenbush & Bryk, 2002) was used to examine the differences in student achievement 
between the treatment and comparison groups and to account for the nesting of students in 
schools. The study was approved by Sterling IRB in 2014. 


2.1 Selection of Schools, Teachers, and Students 
2.1.1 School Selection 


Four schools from DeKalb County School District (DCSD) near Atlanta were purposively 
selected for participation in this study. Conversations with DCSD and community partners 
helped to identify the local schools and communities that would benefit most from additional 
family engagement and school transition supports. DCSD recommended two different feeder 
patterns to target (Clarkston and Stone Mountain), and within those clusters, we identified 
appropriate schools using the following criteria, with preference given to higher levels of 
need: 1) percent economic disadvantage, 2) percent Limited English Proficiency, 3) Title 1 


National Evaluation of i3 05/16/2016 
6 


status, 4) percent below proficient on ELA standards, and 5) percent below proficient on 
reading standards. Using these criteria, as well as securing principal buy-in, Allgood 
Elementary and Dunaire Elementary were selected from the Clarkston High School feeder 
pattern, and Stone Mill Elementary and Stone Mountain Elementary were selected from the 
Stone Mountain High School feeder pattern. These schools were then randomly assigned as 
implementation sites (Allgood Elementary, Stone Mill Elementary) and comparison sites 
(Dunaire Elementary, Stone Mountain Elementary). Early childcare and early education 
centers known to feed into the selected elementary schools were targeted for pre-K 
participant recruitment. Once recruitment was completed, early childcare and early 
education centers were classified as either treatment or comparison sites depending on 
where the majority of participants intended to enroll in elementary school (Table 2). 


Recruitment for cohort 1 began in September 2014 and, due to lower than expected 
recruitment, additional early education centers were added as comparison sites based on 


the feeder school pattern to increase power for the analysis (see Table 2 below). 


Table 2. Study Schools 


School/Center Condition Assignment Grade Levels 
All Star Kids Academy | Treatment Pre-K 
Allgood Elementary School Treatment Pre-K - Grade 5 
Appletree Learning Center* | Comparison Pre-K 
Busy Bee Childcare and Learning 

Comparison Pre-K 

Center* 
Developing Minds* Comparison Pre-K 
Dunaire Elementary School Comparison Pre-K - Grade 5 
Early Learning Scholars Treatment Pre-K 
Good Foundation Treatment | Pre-K 
Hambrick Head Start* Treatment Pre-K 
Journey to Wisdom Comparison Pre-K 
Mem’s Daycare Comparison Pre-K 
Partnership for Community Comparison Pre-K 
Action/Lawrenceville Head Start 
Powell Early Learning* | Comparison | Pre-K 
Stone Mill Elementary School | Treatment | Pre-K - Grade 5 
Stone Mountain Elementary School Comparison PreK - Grade 5 
Teach O’Rea Treatment Pre-K 
Good Foundation | Treatment Pre-K 


*Early education center added in 2015 to increase sample size for the Comparison group 
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2.1.2 Selection of Study Teachers 


All pre-K through Grade 3 teachers working at the treatment elementary schools and early 
childcare centers were included in the study. Teachers at the treatment schools and centers 
only interacted with students in the treatment condition. Teachers in the comparison 
elementary schools and early childcare centers only interacted with students in the 
comparison condition. 


To meet the needs of the student participants in the Summer Transition Academy, teachers 
from Allgood Elementary and Stone Mill Elementary - the two treatment elementary schools 
- were invited to apply for the STA teaching position. Applications were reviewed based on 
teaching history at the targeted grade level, experience teaching literacy skills, and school 
principal recommendations. STA teacher recruitment took place yearly from January to 
April. Sixteen elementary school teachers from the treatment elementary schools were 
selected to participate each year. 


2.1.3 Selection of Students 


All families who had at least one age-eligible child enrolled in the relevant grades at 
participating schools were eligible to join the study during each recruitment period 
(September through December). Although the condition was assigned at the school level, 
families in both the treatment and comparison conditions were required to opt-in to the 
study by completing an enrollment form and participating in data collection. 


Across all sites, the inclusion criteria for participating families was the same. Families were 
eligible to join the study if they had at least one child between the ages of 3-7 and enrolled 
in pre-K to Grade 2 in one of the treatment or comparison sites. After initial recruitment, a 
screening checklist was used to confirm that participating families spoke either English or 
Spanish. 


Families with children who attended the relevant grades at Allgood Elementary, Stone Mill 
Elementary, and the early childcare centers that feed into them were identified as treatment 
families and received PACE services. PACE families received a combination of monthly 
personal visits, participated in parent group meetings, and were exposed to the School 
Transition Council and the Summer Transition Academy. Teachers and early childcare 
educators working at each of the treatment sites received professional development 
workshops on parent engagement and school readiness, as well as invitations to attend 
quarterly School Transition Council meetings. 
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Families with children enrolled in the relevant grades who attended Dunaire Elementary, 
Stone Mountain Elementary, and the early childcare centers that feed into them were 
identified as comparison families and did not receive any PACE services. Comparison 
teachers and early childcare educators did not receive any PACE services. Comparison 
families, teachers, and early childcare educators only participated in pre-and post-test data 
collection and were exposed to traditional, academic-based school environments. 


Three cohorts participated in this study across 4 years, 8 months. This study was originally 
intended to last 4 years and recruit two cohorts of 1,000 children in total (500 PACE and 500 
comparison children). However, due to lower recruitment, higher attrition, and a later start 
than expected, the study framework was altered to include an additional cohort of students 
to increase sample size and power to detect differences across groups. Cohort 1 families 
received services from the fall of 2014 through the fall of 2016 and included 178 participants, 
of which 139 were PACE children and 39 were comparison children. Cohort 2 was added to 
increase sample size to address lower recruitment and higher loss to follow-up than 
expected in cohort 1. Cohort 2 families received services from the fall of 2015 through the 
fall of 2017 and included 209 children of which 121 were PACE and 88 were comparison. 
Cohort 3 families received services from the fall of 2016 through the summer of 2018 and 
included 298 children of which 130 were PACE and 168 were comparison children. Over the 
course of three cohorts, PACE recruited 685 children (68.5% of target) of which 390 were 
PACE children (78% of target) and 295 were comparison children (59% of target). 


2.2 Data Collection 


The data collection timeline was relatively consistent across cohorts (Table 3). For each 
cohort, pretest student, parent, and education outcomes were measured in months 1-4, 1- 
year posttests were collected in months 12-15, and 2-year posttests were collected in 
months 21-24. However, 2-year posttests were not collected for cohort 3, as data could not 
be collected prior to the end of the grant. Due to retention challenges and the funding period 
of the no-cost extension ending prior to cohort 3 post-2 data collection, the current impact 
study examines the results of PACE programming after 1 year of exposure (post-1 data 
collection; Table 3). 


Table 3. Data Collection Timeline 


Pretest 1-Year Posttest _ 2-Year Posttest 
Cohort 1 Sept - Oct 2014 | Sept - Oct 2015 Sept - Dec 2016 | 
Cohort 2 Sept - Dec 2015 Sept - Dec 2016 Sept - Dec 2017 
Cohort3 Sept - Dec 2016 _ Sept-Dec2016 Not collected | 
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Outcome data for each domain were collected using valid and reliable measures that were 


consistently measured across Treatment and Comparison groups (Table 4). 


Table 4. Outcome Measures Psychometric Properties 


Domain 


Literacy 


Language 


Mathematics 


Social- 
Emotional 


Test 


Bracken School Readiness 
Assessment - Letters 
Subtest 


Iowa Test of Basic Skills - 
Reading Subtest 


Measures of Academic 
Progress - Literacy 
Peabody Picture 
Vocabulary Test (PPVT) 
Iowa Test of Basic Skills - 
English Language Arts 
Subtest 
Measures of Academic 
Progress - Language 
Bracken School Readiness 
Assessment —- Numbers/ 
Counting, Comparison/ 
Size, and Shapes Subtests 


Iowa Test of Basic Skills - 
Mathematics Subtest 


Measures of Academic 
Progress - Mathematics 


Social Skills Improvement 
System (SSIS) - Parent 
Form 


Internal 
Consistency 


0.95 


Grade 1 = 0.95 
Grade 2 = 0.93 
Grade 3 = 0.94 


Not available 


0.94-0.95 


Grade 1 = 0.79 
Grade 2 = 0.87 
Grade 3 = 0.88 


Not available 


0.95 


Grade 1 = 0.81 
Grade 2 = 0.92 
Grade 3 = 0.93 


Not available 


0.95 


Test-Retest 
Reliability 


0.86 


Not available 


0.87 


0.93 


Grade 1= 0.76 
Grade 2= 0.81 
Grade 3 = 0.83 


0.89 


0.86 


Grade 1 = 0.82 
Grade 2 = Not 
available 
Grade 3= Not 
Available 


0.83 


0.86 


Marginal 
Reliability 


Not available 


Not available 


0.95 


Not available 


Not available 


0.94 


Not available 


Not available 


0.92 


Not available 


The data collector and the standardized test used to measure outcome domains differed by 
cohort, domain, and age of the child participant (Table 5). Treatment and comparison 
participants in kindergarten or pre-K were assessed by PACE program staff with the 
assistance of The Evaluation Group (TEG) staff on literacy, language, and mathematics via 
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the Bracken School Readiness Assessment - Letters Subtest, Peabody Picture Vocabulary 
Test (PPVT), and Bracken School Readiness Assessment - Numbers/Counting, 
Size/Comparison, and Shapes Subtests, respectively (Table 6). 


For Treatment and comparison participants in Grades 1 and higher, academic outcome data 
were measured through standardized tests conducted by DeKalb County School District. In 
2014 and 2015, students’ academic achievement was measured with the Iowa Test of Basic 
Skills (ITBS) exam. Based on the DeKalb County School District schedule, students in second 
grade did not take the ITBS and therefore, there is no academic outcome or baseline data for 
students enrolled in second grade for these years. In 2016, DeKalb County School District 
shifted to the Measures of Academic Progress (MAP) exam and MAP data were collected for 
children in Grades K-5 so there is no systematic missing data by grade as there was for the 
IOWA test. However, MAP language data was not available for all participants, and therefore, 
participants who enrolled in kindergarten and had reached Grade 1 by posttest were also 
given the PPVT to measure language growth. 


For all participants, social emotional development was tracked via the Social Skills 
Improvement System (SSIS) - Parent Form and there is no systematic missing data because 
the data were collected solely by PACE staff with the support of TEG. 
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Table 5. Data Collection by Cohort 


Cohort 1 Cohort 3 
Grant YR1 : 7-year 
3-year 4-year 5-year 6-year lie 
2014/2015 olds olds olds olds (Grade) 
(Cohort 1 (Pre-K1) | (Pre-K2) (K) (Grade 1) arate 
Pretest) Pretest Pretest Pretest Pretest available 
L L Lo L 4 
Grant YR2 p " : A , Z 
-year -year -year -year 6-year -year 
2015/2016 ais Sear ate olds veal ate Ae aoe aes de 
(Cohort 1 olds olds San olds olds ; 
(Grade (Grade2) ,,. | 3. (Pre- (Pre- (Grade (Grade 2) 
Post1; (Pre-K) (K) (Grade 3) (K) 
Post1 Postl 1) Data not Bact K1) K2) pre 1) Data not 
Cohort 2 “ He Post1 | available aS Pretest Pretest eres’ Pretest _ available 
Pretest) 
L L L L J L L L L L 
Grant YR3 
2016/2017 P : 5 j 
(Cohort 1 5 years ae se 8-year 9-year ee 5-year cae ae 8-year 3-year | 4-year ‘5-year ee ae 
Post2; olds (Grade ( eee olds olds (Pre. olds ( ae (emde olds olds olds olds (eden iGraue 
Cohort 2 (K) 0 2) (Grade 3) (Grade 4) K) (K) 1) 2) (Grade 3) (Pre-K1) (Pre-K2) (K) 1) 2) 
ae Post2 Post2 Post2 Post2 Post2 Boch Post1 Postl Becet Post1 Pretest Pretest Pretest Peekese uPreicae 
ohort 
Pretest) 
L L L L L L L L L L 
Grant YR4 E 5 Z : 5 
2017/2018 -year ~year -year : ‘ -year -year -year 
(Cc ee 2 ; os olds olds olds ae ie aes olds olds olds 
Grade Grade Grade Grade Grade Grade 
Post2; | a | ee ee ‘9 | 3 | 3 
aay ve 2 ae Post1 Post1 Post1 
ost 
L L L L L 
Post grant pee O-year 7-year 8-year 9-year 
period Z ae olds olds olds olds 
2018 (Grade (Grade (Grade (Grade 
(K) 
(Cohort 3 Post2 1) 2) 3) 4) 
Post2) Post2 Post2 Post2 Post2 
Notes: Orange cells indicate PACE team collects academic data (Bracken, PPVT). Blue cells indicate district collects academic data (IOWA). Purple cells indicate no academic data collected. 
Green cells indicate district collects academic data (MAP test). Grey cells indicate cohort ineligible for data collection. Social Emotional Data collected by PACE study team all grant years. 
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Table 6. Measures by Cohort and Grade 


Cohort 1 


Cohort 2 


Cohort 3 


Pretest-Fall 


Posttest-Fall 


Pretest-Fall 


Posttest-Fall 


Pretest-Fall 


Posttest-Fall 


2014 2015 2015 2016 2016 2017 | 
BSRA-Letters BSRA-Letters or 
1. = = a 
. PreK BSRA!-Letters BSRA-Letters BSRA-Letters MAP®-Literacy BSRA-Letters MAP-Literacy 
-) ; BSRA-Letters or BSRA-Lettersor | BSRA-Letters or 
- i ‘ 
5 K BSRA-Letters ITBS-Reading BSRA-Letters MAP-Literacy MAP-Literacy MAP-Literacy 
— 
ca ITBS2- ; ; 
ml : = s = 
1 Reading ITBS-Reading MAP-Literacy MAP-Literacy MAP-Literacy 
Z ITBS-Reading MAP-Literacy MAP-Literacy MAP-Literacy 
PPVT or PPVT or 
3 
0 PreK PPVT PPVT PPVT MAP-Language PPVT MAP-Language 
© 
5 : PPVT or PPVT or PPVT or 
oe . Feve sail pete MAP-Language MAP-Language MAP-Language 
= 1 ITBS-ELA ITBS-ELA MAP-Language | MAP-Language MAP-Language 
2 ITBS-ELA MAP-Language | MAP-Language MAP-Language 
PreK BSRA-Math | BSRA-Math BSRA-Math BSRA-Math BSRA-Math Serial 
MAP-Math 
= 
rm BSRA-Math or BSRA-Math or BSRA-Math or 
S K BSRA-Math ITBS-Math BSRA-Math MAP- Math MAP- Math MAP- Math 
1 ITBS-Math ITBS-Math MAP- Math MAP- Math MAP- Math 
Z ITBS-Math MAP- Math MAP- Math MAP- Math 
7 | PreK SSIS* SSIS SSIS SSIS SSIS SSIS 
"Ss § K SSIS SSIS SSIS SSIS SSIS SSIS 
— 
3 = 1 SSIS SSIS SSIS SSIS SSIS SSIS 
- 2 SSIS SSIS SSIS SSIS SSIS SSIS 


Notes: Beginning in 2016, participants might have taken multiple pretests or posttests for the math, language, and literacy 
domains due to changes in the DeKalb County Schools testing calendar. Data paths were prioritized to favor similar pretest 
and posttest combinations. 
1 BRSA - Bracken School Readiness Assessment 2 ITBS - Iowa Test of Basic Skills ? PPVT - Peabody Picture Vocabulary Test + 
SSIS- Social Skills Improvement System ° MAP - Measures of Academic Progress 
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3. ANALYSIS AND RESULTS 
3.1 Analytic Approach 


The PACE evaluation examined differences in student outcomes between a treatment group 
of PACE students who had participated in PACE over the course of their first year in the PACE 
program (i.e., had at least two personal visits, one group connection, one day of Summer 
Transition Academy, or one day of Family Leadership Institute) and a matched comparison 
group (i.e., business as usual). Student performance data on developmental assessments 
(Bracken School Readiness Assessment, Peabody Picture Vocabulary Test, Social Skills 
Inventory Scale) and standardized test scores (Iowa Test of Basic Skills and Measures of 
Academic Progress) were used in combination with propensity score weights to establish 
baseline equivalence separately for each research question. Details on baseline equivalency 
testing are described in section 3.3 below. 


Propensity score matching was used to predict the selection of families into the PACE 
program. The calculated propensity scores were incorporated as weights into multi-level 
analyses to test each confirmatory contrast proposed in this study. Details on propensity 
scores are described in section 3.5 below. 


As mentioned above in Table 6, participants were assessed on each outcome domain 
(literacy, language, and math) using different measures for different grade levels (PreK 
through 3). To address the use of multiple tests to examine growth in mathematics, literacy, 
and language, assessment scores were standardized into z-scores so all grade levels could be 
pooled into the same analysis. Using this approach, students had a score in each outcome 
domain that was measured on the same scale, regardless of their grade level. Within each 
domain, the z-scores were calculated separately by participants’ grade at enrollment (i.e., 
pre-K, kindergarten, and Grades 1 and 2). For some tests, both the raw and age/gender 
standardized scores were available. Where possible, the raw score was used as the basis for 
the z-score calculation; however, the RIT or standardized scores were also used when the 
raw score was not available (Table 7). We did not calculate z-scores for the SSIS, because the 
same measure was used for all grade levels. All grade levels were pooled into the same 
analysis using the SSIS standard scores. 


Table 7. Scores Used to Calculate the Z-Score, By Test 


peetson Social Skills | lowa- Reading, sae 


BRSA- __BRSA- Picture Language, 
Improvement Language or 
Letters Numbers Vocabulary System (SSIS) Math math or 
Test (ppvty OP reading _ 
Raw Raw Standard Standard Raw Raw Raw RIT RIT RIT 
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Further, due to changes in the DeKalb County School District’s testing calendar, multiple 
pretests and posttests were collected on some participants for a single domain. For example, 
a cohort 2 student enrolled in first grade who had not yet reached 7 years old could have 
been assessed for reading growth via the BRSA and the IOWA reading subtest. To ensure that 
students’ data were not counted twice, data paths were prioritized for the analysis. Priority 
was assigned to those paths that had the same pretest and posttest format (e.g., MAP- 
Language to MAP-Language) with the exception of the BRSA. BRSA data were found to have 
a ceiling effect and were therefore considered lower priority for the outcome data paths. 


Given that multiple community tracks are used within this study (Clarkston: Allgood 
Elementary and Dunaire Elementary; Stone Mountain: Stone Mill Elementary and Stone 
Mountain Elementary), multi-level models were used to estimate student impacts. All three 
cohorts were combined into one model per research question and participants were nested 
in the school they attended at baseline measurement. 


3.2 Sample 


Participants included in the analysis had complete baseline and posttest data and had 
complete demographic data (i.e., age, sex, race/ethnicity, and language spoken at home; 
Table 8). Participants were also included in the analysis if they had participated in i3 PACE 
(e.g., at least two personal visits, one group connection, one day of Summer Transition 
Academy, or one day of Family Leadership Institute). These cases were selected to geta more 
accurate estimate of the impact of PACE for participants who received more than one quick 
and short component of the PACE program. Regarding missing data, we utilized list-wise 
deletion if data were missing on variables of interest. 


Table 8. Analysis Sample, by school _ 


Participants ysctuded in the 
School/Center Grades oe Impact Analysis, 
___ Cohorts 1-3 
Allgood Elementary School PreK - Grade 3 132 89 
Stone Mill Elementary PreK - Grade 3 155 | 110 
= All Star Kids Academy PreK | 14 | 5 
= Early Learning Scholars PreK | 1 1 
$ Good Foundation PreK 28 14 
E Hambrick Head Start PreK 3 0 
| Teach O’Rea PreK 48 | 30 
Total 388 249 
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Participants 


Participants : 
School/Center Grades Recruited, Miclugea™ me 
Cohorts 1-3 Impact Analysis, 
ee a res |e - ___ Cohorts 1-3 
Dunaire Elementary School PreK - Grade 3 114 SS 
Stone Mountain Elementary preR’< Grades 92 35 
School 
2 3 
5 Memorial Drive PreK 1 0 
Presbyterian | 
& Journey to Wisdom PreK 2 0 
E Partnership for Community PreK 74 35 
Oo Action | 
Developing Minds PreK 16 8 
Appletree Learning Center PreK 6 3 
TOTAL 295 156 


Total number of participants differed by research question based on available baseline and 
post-test data (Table 9). 


Table 9. Analysis Sample, by Domain 


Number of Number of Number of Number of 
Domain Treatment Comparison Treatment Comparison 
= ___ Students __—Students__—Schools__Schools 
Reading 166 Ara ke) 5 
Language | 121 | 44 5 5 
Mathematics | 168 | 119 5 | 5 
Social-Emotional 163 17 5 5 


3.3 Baseline Equivalence 


We tested for the equivalence of students who did and did not receive PACE using a pretest 
measure for the dependent variable. Note that we analyzed baseline equivalency separately 
for each of our research questions. Groups were considered equivalent at baseline if the 
effect size difference between PACE and non-PACE students was less than 0.25 and the 
baseline measure was included in the analytic model. All dependent variables met baseline 
equivalence on a pretest measure. 


Additionally, Hedge’s g was calculated: (math (ES = -0.03), reading (ES = -0.05), language (ES 
= 0.03), and SSIS (ES = -0.10)). 
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3.4 Confirmatory Questions 


Confirmatory Question 1: What is the impact of PACE on students' math achievement 1 year 
after entering PACE compared to students in a business-as-usual comparison group? 


Confirmatory Question 2: What is the impact of PACE on students' reading achievement 1 
year after entering PACE compared to students in a business-as-usual comparison group? 


Confirmatory Question 3: What is the impact of PACE on students' language achievement 1 
year after entering PACE compared to students in a business-as-usual comparison group? 


Confirmatory Question 4: What is the impact of PACE on students' social skills 1 year after 
entering PACE compared to students in a business-as-usual comparison group? 


3.5 Propensity Score Model 


To estimate a propensity score for each child, we fit a series of random-intercepts logistic- 
regression models to predict the selection of families into the PACE program (versus non- 
PACE). The following covariates were included in the calculation of the propensity score: 
pretest scores on literacy, language, mathematics, and social-emotional measures, student 
sex (female or male), age at entry into PACE, ethnicity (non-minority or minority) as well as 
primary language spoken in the home (English and Spanish/other). The hypothesized 
selection model therefore takes the following form for the i child in the j setting: 


P{PACE,; |V,X} A 4 gn(Ba y+ Bay) 
Where P{ } represents the probability that a child would enroll in a PACE program, and vector 
V and X, represent the child and household covariates respectively. 


3.6 Confirmatory Analytic Model 


The multi-level model we used to estimate a treatment effect for language, literacy, 
mathematics, and social emotional development after 1 year of program exposure is shown 
below. Note that the same model specification was used for each outcome, so the model was 
run a total of four times. 


Yjj = « + Baseline;B; + Condition;B, + EnrollmentAge;B3; + Gender\;B4 
+ OtherRaceDummy;jB; + HispanicRaceDummy;;B, + Cohort2Dummy;;B7 
+ Cohort3DummyjjBg + € + ej 
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Where: 


Yj= the outcome for student i in school j 

a = intercept 

Baseline,;B, = parameter estimate for the effect of the baseline or pretest score 
Condition;62 = treatment effect, or treatment comparison difference (1 = treatment and 0 
= comparison) 

EnrollmentAge;;B3 = effect of student enrollment age (mean centered, higher numbers 
indicate older age) 

Gender;jB, = effect of student gender (1 = female and 0 = male) 

OtherRaceDummy;jf; = race dummy variable 1 (1 = Other race, 0 = Not other) 
HispanicRaceDummyjjB¢ = race dummy variable 2 (1 = Hispanic, 0 = Not Hispanic) 
Cohort2Dummy,;B7 = cohort dummy variable 1 (1 = cohort 1, 0 = not cohort 1) 
Cohort3Dummy;;Bg = cohort dummy variable 2 (1 = cohort 2, 0 = not cohort 2) 

€; = arandom error term for school j 

ej =a random error term for student i in school j 


Sensitivity analyses were also performed to examine if a more complex model that 
incorporated the pretest and posttest combinations across students was a better fit for the 
data. Since the results produced were the same across models, the more parsimonious 
models were selected as described above. 


3.7. _—_- Results for Math Achievement Outcome 

Results indicated no statistically significant difference between the PACE treatment group 
and the business-as-usual comparison group on the math achievement outcome (Table 10). 
No statistically significant findings were found in our model, except for baseline math 


achievement scores positively predicting posttest math achievement scores. 


Table 10. PACE Math Achievement Model 


Variable Estimate Standard t-value p-value 
| Error | | 
Intercept “077 0.16 | =2:9 1 0.004 
Math Achievement 0.70 0.04 15.80 < 0.001 
Baseline Score | | | 
Condition 0.14 | 0.10 L 1.34 | 0.218 
Other 0.21 0.14 1.51 0.133 
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Standard 


Variable Estimate t-value p-value 
Error 
Hispanic 0.00 0.18 0.01 0.991 
Enrollment Age 0.08 0.04 1.94 0.053 
Gender 0.04 0.09 0.48 0.632 
Cohort 1 0.26 0.15 1.69 0.092 
Cohort 2 0.27 | 0.15 1.83 0.068 


3.8 Results for Reading Achievement Outcome 


Results indicated no statistically significant difference between the PACE treatment group 
and the business-as-usual comparison group on the reading achievement outcome. No 
statistically significant findings were found in our model, except for baseline reading 
achievement scores positively predicting posttest reading achievement scores. Table 11 
includes our regression model output. 


Table 11. PACE Reading Achievement Model 
Standard 


Variable Estimate t-value p-value 
—— | Error — EC — 

Intercept 0.03 | 0.27 0.10 0.920 
ae ao 0.50 0.06 9.03 < 0.001 
Condition -0.19 0.27 -0.71 0.499 
Enrollment Age -0.06 0.06 -1.02 0.309 
Other 0.08 0.17 0.45 0.650 
Hispanic -0.21 0.22 -0.949 0.344 
Gender 0.06 0.10 0.59 0.554 
Cohort 1 -0.06 0.19 -0.34 0.735 
Cohort 2 -0.12 0.18 -0.69 0.493 


3.9 Results for Language Achievement Outcome 


Results indicated no statistically significant difference between the PACE treatment group 
and the business-as-usual comparison group on the language achievement outcome. Two 
statistically significant findings were found in our model: higher pre-measure language 
achievement scores predicted higher outcome language achievement scores and higher age 
at enrollment predicted higher outcome language achievement scores. Table 12 includes our 
regression model output. 
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Table 12. PACE Language Achievement Model 


Variable Estimate slandard t-value p-value 
Error 

Intercept | -0.01 0.22 -0.03 0.977 
a aaa 0.74 0.06 12.13 < 0.001 
Condition iu) 018  -095  — 0.371 
Enrollment Age 0.15 0.06 2.46 0.015 
Other 0.17 0.20 0.84 0.400 
Hispanic 0.00 0.25 0.00 0.998 
Gender / 0,09 0.12 0.76 0.450 
Cohort 1 0.09 0.17 0.51 0.608 
Cohort 2 0.21 0.17 1.28 0.202 


3.10 Results for Social Skills Outcome 


Results indicated no statistically significant difference between the PACE treatment group 
and the business-as-usual comparison group on the social skills outcome. Only one 
statistically significant finding was found in our model: higher pre-measure social skills 
scores predicted higher outcome social skills scores. Table 13 includes our regression model 
output. 


Table 13. PACE Social Skills Model 
Standard 


Variable Estimate _ t-value p-value 
Error _ 
Intercept 45.62 14.50 3-15 0.002 
aii Ce 0.67 0.10 7.06 < 0.001 
Condition -9.47 10.02 -0.95 0.372 
Enrollment Age 2.01 1.23 1.63 0.104 
Other -6.49 S212 “127 0.207 
Hispanic -2.35 6.54 -0.36 0.720 
Gender -0.14 3.05 -0.05 0.963 
Cohort 1 1.79 4.15 0.43 0.668 
Cohort 2 1.84 4.17 0.44 0.589 


4. DISCUSSION 


The current study applies a quasi-experimental pre-posttest design to examine the impact 
of i3 PACE on reading, language, mathematics, and social emotional outcomes compared to 
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business as usual after 1 year of program exposure. Overall, the study found no statistically 
significant differences in the treatment versus the comparison group for each of the outcome 
domains. 


Our findings were in contrast to two prior evaluations of iterations of the i3 PACE program 
- SPARK and PACE - which found improved outcomes for parents and their children. Those 
studies indicated that we should see gains in pre-K and kindergarten students while gains in 
older elementary school students were more exploratory. 


The results of the current quantitative analysis also do not reflect the results of the 
qualitative studies that the independent evaluation team conducted in 2016 and 2018. In 
2016, two focus group discussions were conducted with parent participants in i3 PACE. The 
parents were selected for their active participation in and knowledge of the i3 PACE program 
from the distinct vantage points as parents with younger (pre-K and kindergarten) and older 
(Grades 1-3) i3 PACE children. From these parents we learned that i3 PACE increased the 
self-confidence of parents and children, encouraged children to read on their own and with 
their parents, and provided parents with key emotional support and community connections 
that they would utilize long after i3 PACE ends. For example, one parent described i3 PACE 
as “an enrichment program for me and my family with all different types of elements to help 
the children be successful academically and for us to be, as a family, successful.” The 
interviews conducted in 2018 with i3 PACE parents and educators from the two i3 PACE 
elementary schools (n=16) also provided insight into i3 PACE achievements. Again, 
participants were selected based on their knowledge of the i3 PACE program and diverse 
experience with its key components. Both educator and parent participants indicated that 
PACE improved students’ educational and social emotional outcomes by helping families and 
educators build stronger relationships and providing families and educators with strategies. 
As one parent participant indicated, “I still look, I still flip through my manual for different 
things and skills like we did a family crest thing and I wanted to do that with the kids and so 
just even to this day, it was two years ago, still bringing back just positive personal memories 
and actual workable skills that I can apply.” One i3 PACE educator similarly stated, “one thing 
that the PACE program taught us is to allow the children to speak. I probably learned that 4 
years ago with PACE, and I've been doing it in my classroom because | realized when you 
allow them to talk freely, they actually stay on task and it's not overwhelming and out of 
control, as long as you go over the rules. So, I did get that from PACE and I use that to this 


y 


day. 


Several limitations of the study design and program implementation may have contributed 
to our null findings. First, implementation of the program key components differed 
significantly across years with a slow start-up that delayed implementation of the School 
Transition Councils until program year 3 (calendar year 2016). Second, the i3 PACE program 
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was composed of multiple curricula and lacked a unifying framework which may have 
decreased implementation fidelity. Third, regarding the study design, we utilized multiple 
tests to examine growth in each domain and were required to calculate z-scores to measure 
growth which likely increased error in our analysis. While the application of z-scores were 
anticipated, changes in DeKalb County Schools testing required that, for each domain, we 
calculate growth using four separate tests. Fourth, regarding the Bracken School Readiness 
Assessment, we found an important ceiling affect for older, eligible children which likely 
blunted our ability to detect differences for children ages 3 to 6 years 11 months. Finally, we 
were unable to control for parental measures of socioeconomic status due to relatively large 
amounts of missing data; however, given the overall homogeneity of the students at the 
elementary schools, missing these data may not have had a large impact. 


The current study also has several strengths that should be leveraged in future studies of 
family engagement programs such as i3 PACE. The study used reliable and valid measures to 
examine participant growth in skills, such as the Social Skills Improvement System. The 
study also sought continuous feedback from participants using mixed methods so that i3 
PACE participants could receive the information and services that they wanted and needed. 
In this regard, the program flexibility may have contributed to the null findings of the current 
study but provided better services to the community. In sum, while i3 PACE demonstrated 
no significant gains in i3 PACE participants compared to business-as-usual comparison 
students, the study has identified several strengths that should be leveraged to inform future 
iterations of the program. 
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